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ABSTRACT 

To debunk the myth that evaluation is a process better left 
to people with a PhD, this guide describes how organizations and coalitions 
can develop and utilize an evaluation strategy to make underage drinking 
prevention programs successful and effective. This booklet first describes 
the purpose of evaluation as well as the myths and facts about the evaluation 
process. For instance, contrary to popular opinion, evaluations do not have 
to be time consuming and expensive to be useful. When evaluation is an 
integral part of the planning and implementation process, it includes four 
stages or types of evaluation that are described in detail, including 
formative, process, outcome and impact. In addition to descriptions, the 
booklet provides examples of when each stage should be used. Proper planning 
of an evaluation is critical to its success and the booklet describes the 
process to be used as well as what to look for if hiring an evaluator. (GCP) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Community lloui To Guide On 













I 

I 




(oinmunitij Hoiu To Guide 0n...EVIlLUIlTI011 

TiLEOfCOnTTOTT 



Executive Summary 

Acknowledgements 

Introduction 

Purpose of Evaluation 

Myths & Facts About Evaluation 

The Four Stages of Evaluation 

Formative Evaluation 

Process Evaluation 

Outcome Evaluation 

Impact Evaluation 

Evaluation Methods 

Quantitative Methods 

Qualitative Methods 

Planning An Evaluation 

Hiring An Evaluator 

DPPtnDiffi 

Appendix #1 - Tracking Form 

Appendix #2 - Evaluation Plan Worksheet 

Appendix #3 - Sample Personal Interview Question 
Appendix #4 - Sample Focus Group Questions . . . 
Appendix #5 - Participant Observation Form .... 
Appendix #6 - Pilot Projects 



. 1 
.2 
.3 
.5 
.5 
.6 
.7 
. 8 
.9 
10 
11 
11 
12 
13 
15 



. iii 

. V 

viii 

, . X 

xiii 



Resources Cited in Community How To Guide 

Other Evaluation Resources 

Evaluation Publications 





4 



Underage Orinkin?Pr«ntion Project 




(ouiiminitij Houi To Guide On...£VfllUflTIOIl 




C ities, counties, and neighbor- 
hoods across America are 
confronting the problem of 
underage drinking and its 
consequences. The National 
Association of Governors’ Highway Safety 
Representatives (NAG HSR) is a professional 
organization representing the chief highway 
safety officers from each state, the District 
of Columbia, and the U.S. territories. 
NAGHSR is committed to prevent and 
reduce illegal underage alcohol consumption 
and to curb the terrible toll underage 
drinking takes on our society. 

As a result of their commitment to underage 
drinking issues and to assist cities, counties, 
and neighborhoods, NAGHSR, with 
financial assistance from the National 
Highway Traffic Safety Administration 
(NHTSA), developed a series of 
“Community How To Guides. ’’These 
“Community How To Guides'* dAAvcss 
fundamental components of planning and 
implementing a comprehensive underage 
drinking prevention program. The Guides 
are designed to be brief, easy-to-read, and 
easy-to-use. Each guide contains a resource 
section to assist readers in obtaining 
additional and detailed information about 
the topics covered in that guide. The 
appendices include useful tools for each 
topic area that provide coalitions and 
organizations a jump-start in their planning 
and implementation activities. 




Topics covered in the “Community 
How To Guides" \nc\\xdt: 

■ Coalition Building 

■ Needs Assessment and Strategic Planning 

■ Evaluation 

■ Prevention and Education 

■ Underage Drinking Enforcement 

■ Public Policy Advocacy 

■ Media Relations 

■ Self-Sufficiency 

■ Resources 

In addition to the “Community How To 
’’NAGHSR also developed a Case 
Study of a model community underage 
drinking prevention program. This study is 
designed to give other communities which 
are considering starting an underage drinking 
prevention program or those that have an 
existing effort, some practical, real-world 
ideas and suggestions. The comprehensive 
approach, set forth in the “how to guides,” 
has been applied in this community to 
great success. 

The impetus for developing these guides 
came from the work NAGHSR has done in 
the area of underage drinking prevention for 
NHTSA and the Office of Juvenile Justice 
and Delinquency Prevention (OJJDP). 

Based on their work in the underage 
drinking prevention arena, NAGHSR felt 
the comprehensive approach, first piloted 
in the Washington, D.C. area, could be of 
benefit to any community seeking to reduce 
underage drinking. 
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Although these guides are targeted at the 
underage drinking issue, the basic process 
and information can be applied by any 
community-based organization striving to 
affect a social problem through changing 
community norms. These guides will also 
prove useful to Safe Communities, Students 
Against Destructive Decisions (SADD) 
chapters, Mothers Against Drunk Driving 
(MADD) organizations, police departments, 
and youth groups belonging to the National 
Organizations for Youth Safety (NOYS). 



Violence prevention organizations may also 
find the information and tools helpful since 
the process to reduce incidents of youth 
violence is the same process detailed in 
these guides. 

Contrary to conventional wisdom, underage 
drinking is not inevitable. It is preventable. 
These "'Community How To Guides” 2iTt 
designed to help individuals or organizations 
to craft programs that will help the young 
people in their communities remain 
alcohol-free. 



Eoiumunitij How To Guide 0n...EVHLyflTI0n 
HEKHOIHLEDGEHlEnTE 



T he production of the nine 

"Community How To Guides” on 
underage drinking prevention 
involved the efforts of a number 
of people who helped develop 
the ideas and suggestions for each publica- 
tion and provided the time to review the 
draft materials. 

The National Highway Traffic Safety 
Administration (NHTSA) wishes to thank 
Barbara Harsha, Executive Director, National 



Association of Governors' Highway Safety 
Representatives (NAGHSR) for her 
leadership in the development of the 
"Community How To Guides” 2nA to Pam 
Beer of PMB Communications and Trina 
Leonard of Leonard Communications, the 
authors of the Guides. Special thanks to 
Nancy Rea, Executive Director of Drawing 
the Line on Underage Alcohol Use in 
Montgomery County, Maryland for her 
expertise and review. 






Underage Drinkin? Prevention Project 



6 






(ommunitij Hou) To Guide On.iVflLUflTIO[l 

IDTRODUaiOO 



T O debunk the myth that evaluation is a 
process better left to people with a 
PhD, the “Community How To Guide 
on Evaluation,” describes how 
organizations and coalitions can 
develop and utilize an evaluation strategy to make 
their programs successful and effective. This 
booklet first describes the purpose of evaluation as 
well as the myths and facts about the evaluation 
process. For instance, contrary to popular 
opinion, evaluations do not have to be time 
consuming and expensive to be useful. 

When evaluation is an integral part of the 
planning and implementation process, it includes 
four stages or types of evaluation that are 
described in detail, including formative, process, 
outcome and impact. In addition to descriptions. 



the booklet provides examples of when each stage 
should be used. There are also two methods that 
can be used - quantitative and qualitative. In 
appendix, the booklet provides samples of both of 
these methods, including a form to be used in 
counting numbers for quantitative methods and 
sample interview and focus group questions and a 
participant observation form for use in qualitative 
methods. 

Proper planning of an evaluation is critical to its 
success and the booklet describes the process to be 
used as well as what to look for if hiring an 
evaluator. After reading the booklet, 
organizations and coalitions will know how to 
integrate an evaluation into their overall plan, 
thereby making their efforts more targeted and 
effective. 
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T he American Heritage dictionary 
defines “evaluation” as: 

“L To ascertain or fix the value or 
worth of. 2. To examine and judge: 
appraise, estimate. 3. To calculate or 
set down the numerical value of: to express 
numerically.” 

When program managers hear the word 
“evaluation”, it often conjures up unpleasant 
images of cold-eyed scientists scrutinizing their 
activities and declaring their program to be a 
failure. In reality, evaluation can be and should 
be an asset for program managers and their 
efforts. Strong, carefully designed evaluation can 
help program managers target their efforts, 
develop efficient materials and programs, make 
mid-course adjustments, if necessary and prove 
their success. Evaluation can transform 
guesswork into certainty and can help an 
organization thrive. 

Well-designed evaluation programs can help 
underage drinking prevention programs 
demonstrate their effectiveness. Underage 
drinking prevention specialists know that many 
adults view underage drinking as an inevitable rite 
of passage for youth. Evaluation can disprove 
that myth and can demonstrate that carefully 
targeted programs do reduce youthful drinking. 



■ Demonstrate the need for a specific 
prevention, enforcement or public policy 
strategy. 

■ Target programs and materials to the 
appropriate audience. 

■ Ascertain whether materials and messages will 
be effective (through pre-testing). 

■ Monitor changes in attitudes, behavior, 
knowledge or beliefs among the target 
audience. 

■ Measure progress toward the programs goals. 

■ Determine whether mid-course changes in 
materials, programs, audiences or delivery 
systems are needed. 

■ Establish an early warning system to detea 
problems that could impede the program. 

■ Uncover unexpected benefits or difficulties in 
design or implementation of the program. 

■ Produce data on which to base future 
programs. 

■ Demonstrate the effectiveness of the program 
to the target population, to the public, to 
others who want to conduct similar programs, 
and to those who fund the program. 

diyTHSiDfflCTSffiOOTEVflLyflTIOn 



Tl 







Many of the people who cringe at the word 
“evaluation” mistakenly believe that evaluation 
begins when a program is nearly finished. That 
kind of evaluation offers very litde useful 
information for the program planners. When 
evaluation is an integral pan of the planning 
process, however, it can help program planners to 
do the following: 



Evaluation is not something that should be an 
afterthought. It is an integral pan of the projects 
overall plan and must start from day one. 
Following are some myths and facts about 
evaluation, which were taken from the booklet, 
*'The Art of Appropriate Evaluation, A Guide 
for Highway Safety Program Managers, ” 
produced by the NHTSA. Information on how 
to obtain this guide is listed in the Resource 
Seaion of this booklet. 



mST COPY AmiMBM 
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Myth: Evaluation involves complex research 

methodologies and is too complicated and 
expensive for community-based programs. 



the future. Other underage drinking prevention 
programs may also benefit from lessons learned 
during an unsuccessful process. 





In an ideal world, every community program 
would have the luxury of retaining a highly skilled 
evaluation specialist dedicated only to their effort. 
In reality, most community programs cannot 
afford and often do not require a highly 
complicated evaluation strategy, unless they 
receive funding from a private or government 
agency that requires a specific evaluation 
approach. 

Community-based programs can develop 
evaluation strategies by taking advantage of the 
information in this booklet and other books that 
are available in bookstores, libraries or on the 
Internet, some of which are listed in the Resource 
Section of this booklet. In many cases, the 
program managers may also be able to solicit 
assistance from a college professor, graduate 
student, health department staff member or other 
individual with expertise in evaluation. 



Myth: Evaluation only points out what is wrong 
and exposing difficulties with a program 
may cause funding sources or the public to 
withdraw their support. 



Just as an evaluation may indicate what is not 
working, it can also point out what is effective. 
Effective strategies can be replicated and shared 
with other similar community-based projects. A 
track record of accomplishments makes a project 
more attractive to funding agencies, public 
officials, and the community. By learning about 
strengths and weaknesses, the coalition or 
organization can improve their efforts over time. 

Myth: Evaluations are a lot of work, time 
consuming and expensive. 

Evaluation does not have to be expensive or 
complicated to be useful. There are evaluations 
that involve little or no cost. For instance, 
contact with a college or university may result in 
identifying a professor or graduate student 
interested in conducting a program evaluation as 
part of a special project or course work. An 
evaluator from the state traffic safety, health or 
substance abuse prevention and treatment 
agencies or someone from a marketing, 
advertising or public relations agency might also 
assist the coalition. 



If evaluation is incorporated into a planning 
process from the outset, it helps guide the process 
so that the result is a success, not a failure. 




Also, most funding agencies and organizations do 
not penalize grantees if they test an approach, 
which has a reasonable chance of success, and 
then determine it has not achieved the desired 
goal. Progress is achieved through testing new 
strategies. Funding sources want to know what 
was learned from the experience and how the 
project managers intend to redirect their efforts in 
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When an evaluation strategy is an integral part of 
planning and implementing a prevention 
program, the evaluation includes four stages or 
types of evaluation: formative, process, outcome 
and impact. The following description of the key 
stages of evaluation has been adapted from The 
National Center for Injury Prevention and 
Control in their book “Demonstrating Your 
Program's Worth, ” Some of the examples have 
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been changed to reflect the needs of an underage 
drinking prevention project. 

Formative Evaltiation 

Formative evaluation is most often used to test 
the appropriateness and effectiveness of project 
materials such as a video, public service 
announcement (PSA) brochure, poster, etc. For 
example, if an organization is planning a public 
information and education campaign aimed at 
underage youth that includes a video, it is critical 
to test the format, message and delivery systems 
prior to spending the money to produce the 
video. To pre-test the video, program organizers 
can bring together a focus group of young people 
and present them with the contents of the video, 
usually in the form of storyboards (a cartoon-like 
representation of the material to be presented). 

If the young people in the focus group respond to 
the videos format and message, planners will 
know their product can be effective and their 
money well spent. In evaluating written 
materials, program planners can determine 
whether the product is appropriate for the target 
populations reading level and ethnic background. 

■ Description 

■ Determines whether program plans, 
procedures, activities, materials, and 
modifications will work as planned. 

■ It is an on-going process that should be 
used throughout the programs existence. 

■ Helps to guide and refine messages, 
program components and activities. 



■ When an existing program: 

■ is being modified 

■ has problems with no obvious solutions 

■ is being used in a new setting, with a 
new population 

■ to target a new problem or behavior 

■ What It Shows 

■ Whether proposed messages are likely to 
reach, to be understood by, and be 
accepted by the target audience (e.g., 
shows strengths and weaknesses of 
proposed written materials). 

■ The best time to introduce a program or 
activity. 

■ Whether the proposed plans and strategies 
are likely to succeed. 

■ How people in the target population get 
information (e.g., which newspapers they 
read or radio stations they listen to). 

■ Individuals the target population respects 
as a spokesperson (e.g., a sports celebrity 
or local preacher). 

■ What program developers may have 
overlooked about materials, strategies, or 
mechanisms for distributing information 
(e.g., that the target population has 
difficulty reaching the location where 
training classes are held). 

■ Why It Is Useful 

■ Shows which aspects of the program are 
likely to succeed and which need 
improvement. 



evaWooiTfnoTt 
ofteoudtoteTt 
tlieapprapfiateneTi 
and effect™ of 



announcement (Mh) 






■ When to Use 

■ During the development of a new 
program. 




■ Allows programs to make revisions before 
the full effort begins. 

■ Maximizes the likelihood that the program 
will succeed. 
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■ If the program is in operation, but having 
unanticipated problems, a formative 
evaluation may help find the cause. 



are working and to make adjustments 
before logistical or administrative 
weaknesses become entrenched. 



■ Measurement Methods 







■ Qualitative methods such as personal 
interviews, focus groups, participant 
observations. 

■ Quantitative methods such as surveys may 
be useful. 

Process Evaluation 

Process evaluation is used to determine whether 
the project or coalition is working effectively, and 
may involve interviews with key members of the 
coalition or organization. 



■ Description 

■ It is the mechanism for testing whether the 
programs procedures for reaching the 
target population are working as planned. 



■ When to Use 

■ As soon as the program begins operation, 
and continuously throughout the life of 
the program. 



■ What It Shows 

■ How well a program is working (e.g., how 
many people are participating or reached 
by the program). More important it shows 
how well the process is working and 
whether there are any snags in the system. 




Why It Is Usefid 

■ Identifies any problems that occur in 
reaching the target population. 

■ Allows programs to evaluate how well their 
plans, procedures, activities, and materials 



■ Can be used to show funding agencies the 
programs level of activity. 

■ Can provide encouragement to 
participants. 

■ If the process evaluation shows some 
problem areas, do some additional 
formative evaluation. 

■ Measurement Methods 

■ Direct contacts with the program (number 
of people who call, visit, etc.). 

■ Indirect contacts (public information and 
education campaigns, media stories, etc.). 

■ Items distributed or collected (brochures, 
bumper stickers, etc.). 

Note: If evaluating the effectiveness of a media 
effort, it is important to determine whether the 
news outlet or news program reaches the 
intended target audiences. Newspaper editorials, 
for instance, may not have a large teen 
readership. This is particularly important if an 
organization is conducting a radio campaign. A 
classical or jazz station format is unlikely to 
attract youth listeners, but probably will reach 
parents. Information on demographics is 
available from the radio station or from 
advertising agencies. In addition, media 
messages must also be culturally relevant and 
take into account any target populations with 
low literacy levels or those where English isnt the 
first language. 

Large communities may also have several 
different weekly newspapers that are delivered to 
different geographic areas. In the evaluation 
process, the project should determine whether the 
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information appeared in one paper or all. This 
is even more important for programs that are 
statewide. A group may want to have a map of 
the target area and place colored push pins to 
indicate where messages were received. 

Outcome Evaluation 

Outcome evaluation is used to determine the 
organizations progress toward achieving their 
goals and objectives. Assessing whether there has 
been an increase in the number of stories in 
newspapers and on radio and television that have 
raised public awareness of the seriousness of the 
underage drinking problem is an example of an 
outcome evaluation. 

■ Description 

■ There are several definitions of outcome 
and impact evaluation. For purposes of 
this booklet, outcome evaluation is being 
used to measure the programs progress 
toward immediate or intermediate 
progress. 

■ It measures baseline knowledge, attitudes 
and beliefs of the target population and 
demonstrates changes following 
completion or implementation of the 
program. 



■ When to Use 

■ After the program has begun its activities 
and when contact has been made with the 
target population. 

■ After a baseline measurement has been 
established. (The How To Guide on 
Needs Assessment and Strategic Planning 
describes in depth how to establish a 
baseline measurement. The baseline, 
usually done at the beginning of a 
program, determines the nature and extent 
of the underage drinking problem and 
provides a profile of the community at a 
given point in time.) 

■ What It Shows 

■ The degree to which there are changes in 
the target populations knowledge, 
attitudes and beliefs (e.g., how awareness 
about a party patrol enforcement program 
has changed the attitudes of underage 
youth about the consequences of underage 
drinking). 

■ Why It Is Usefiil 

■ Allows management to modify materials or 
move resources from a nonproductive to a 
productive area of the program. 




■ According to the booklet. Understanding 
Evaluation: The Way to Better 
Prevention Programs^ from the U.S. 
Department of Education, “Outcome 
evaluation assesses program achievements 
and effects and studies the immediate or 
direct effects of the program on 
participants. 



■ Tells program whether they are moving 
toward achieving their goals. 

■ Results can be used to justify the 
continuation of the program. 

■ Measurement Methods 

■ Surveys (see page 1 1 and the Community 
How To Guide on Needs Assessment and 
Strategic Planning 
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■ Observation of people participating in the 
program or project 

Impact Evaluation 

Reviewing data on drinking and driving by 
minors five years after an underage drinking 
prevention program begins, and comparing that 
data to a baseline is one example of an impact 
evaluation. 

■ Description 

■ According to the booklet, Understanding 
Evaluation: The Way to Better 
Prevention^ “Impact evaluation looks 
beyond the immediate results of policies, 
instructions or services to identify longer 
term as well as unintended program 
effects. It may also examine what happens 
when several programs happen in unison.” 

■ Impact (actual reductions in crashes, 
violations, etc.) may not become apparent 
for a long time. 

■ When to Use 

■ The design of an impact evaluation begins 
when the program is being planned to 
insure the right information is collected 
throughout the duration of the program. 

■ When there has been a sufficient amount 
of time for data changes to be significant 
and when the number of people 
participating or reached by the project is 
large enough to be statistically relevant. 

■ After a baseline has been established (see 
explanation under Outcome Evaluation). 



■ What It Shows 

■ The degree to which the program has met 
its ultimate goal (e.g., reduce underage 
drinking). 

■ Why It Is Usefiil 

■ Allows programs to learn from their 
successes and failures and to incorporate 
what they have learned into their current 
or next project. 

■ Provides evidence of success for use in 
future requests for funding. 

■ Measurement Methods 

■ Data collection (See data checklist in the 
Community How To Guide on Needs 
Assessment and Strategic Planning 

Note: Although a decrease in the number of 
underage drinking-related incidents may be 
evidence of program success, impact may not 
be evident in two to three years, usually 
because the sample numbers are too small in 
that period of time. A steady decline over a 
period of years is needed in order to 
determine impact. 
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Quantitative Methods 

Quantitative methods involve data gathering and 
counting numbers and may be used during 
process, outcome or impact evaluation. These 
methods include the following: 

■ Tracking the Program 

This approach involves keeping track of 
whatever the program is evaluating, which 
may include such things as: 

■ Counting the number of people contacted 
or reached by the project. 

■ Counting the number of materials 
distributed. 

■ Counting the number of training sessions 
held. 

■ Counting the number of DWI arrests, 
fatalities, crashes, etc. 

In this guide. Appendix #1 is a Tracking Form 
to assist communities with this approach. 

Note: When counting the number of people 
reached by a media message that appears in a 
newspaper, take the circulation number and 
multiply it by 2. This is usually acceptable for 
most mainstream publications. For broadcast, 
radio and television stations and advertising 
agencies have information on the demographics 
for each program and even for various segments 
of programs (drive-time, second quarter of the 
news hour). 

■ Surveys 

Surveys can be conducted on the telephone, in 
person, or the individual can complete the 
survey in private and deliver or mail it back to 
the organization conducting the evaluation. 
Surveys have a number of uses including: 



■ Gathering baseline data on the knowledge, 
attitudes, beliefs and behaviors of the 
target populations. 

■ Determining changes in knowledge, 
attitudes, beliefs and behaviors. 

■ Helping organizations to modify aspects of 
their program that are not effective. 

■ Measuring the community’s or 
participants’ understanding of the program. 

■ Helping to identify the source of a 
problem. 

■ Measuring the effect of the program. 

The needs and budget of the organization and 
the objective of the evaluation determine the 
type of survey. Surveys people answer in 
person, for instance, which have the highest 
response rate, usually require a trained 
interviewer and are therefore costly. 

Telephone interviews are the quickest to 
conduct and are easily randomized, and 
mailed or delivered surveys offer the greatest 
amount of anonymity. 

In designing survey instruments, use the 
following steps: 

■ Define the population. 

■ Choose the method (imperson surveys, 
telephone interviews, mailed/delivered 
surveys) 

■ Give the survey an appropriate title that 
will generally indicate its purpose. 

■ Put the items in the correct order with the 
least sensitive items (age, location of 
residence, etc.) at the beginning and 
gradually build to more sensitive items. 

■ Pilot-test the survey to determine whether 
it is a valid, reliable instrument and 
questions are not misleading or confusing. 
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The Community How To Guide on Needs 
Assessment and Strategic Planning has two 
sample surveys, a household survey of adults and 
a survey for underage youth. The Pacific Institute 
for Research and Evaluation (PIRE) has also 
produced a booklet entitled ''Guide to 
Conducting Youth Surveys** 2 s part of the Office 
of Juvenile Justice and Delinquency Preventions 
Enforcing Underage Drinking Laws Program. 

This guide includes several examples and 
information on how to structure and conduct 
youth surveys. (See the Resource Section of this 
booklet for contact information.) 

Every two years, NHTSA conducts a National 
Survey of Impaired Driving Attitudes and 
Behavior, which is administered to a random 
national sample of the driving public, age 16 and 
older. The data is reported by age and asks such 
questions as how much people drink and drive, 
what are their views on the problem of drinking 
and driving and how they feel about enforcement 
of impaired driving. Information on obtaining a 
copy of this survey is included in the Resource 
Section of this booklet. 



Since 1991, the Centers for Disease Control has 
also conducted a biennial national school-based 
survey called the Youth Risk Behavioral Survey 
(YRBS) to assess the prevalence of health risk 
behaviors among high school students. National, 
state, territorial and local data is available on such 
highway safety topics as seat belt and bicycle 
helmet use, riding with an impaired driver and 
alcohol and other drug use. 



I? 




Qualitative Methods 

Qualitative methods are generally open-ended 
and are another method to determine a persons 
attitudes, knowledge level and beliefs. These 
methods may be used during the formative and 
process evaluation stages of an evaluation. They 



Underage Drinkin? Prevention Project 



can also help a coalition or organization correct a 

problem if one arises. 

■ Personal Interviews 

Personal interviews are one of the primary 
ways that reporters gather information for a 
news story. Qualitative personal interviews are 
different than an in-person response to a 
survey, since a survey is a standard set of 
questions. Personal interviews are more like a 
discussion where the interviewer asks 
questions to obtain the desired information. 

In a personal interview, the interviewer should 
not interject his or her beliefs or feelings into 
the discussion, but remain neutral to obtain the 
most accurate picture of the interviewees point 
of view. The interviewee is asked to be a 
partner with the coalition or organization and to 
assist them in developing an effective program. 
The larger and more diverse the group of people 
who are interviewed, the better the results. 
Depending on the level of sophistication in the 
evaluation, personal interviews should be taped 
and transcribed verbatim, with the interviewees 
permission. This allows an outside evaluator to 
analyze the results and then provide a written 
report on what was learned. 

■ Focus Groups 

Focus groups are gatherings of up to ten 
participants who are representative of the 
larger target audience. They are brought 
together in a forum to discuss their views 
under the leadership of a trained moderator. 
Focus groups are widely used by marketing 
and advertising research firms to obtain 
insights into target audience perceptions and 
beliefs early in a program or campaign. The 
group atmosphere provides greater stimulation 
and richness of information than can be 
obtained through individual interviews. 
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Focus group interviews provide insight not 
only into what is preferred in a specific 
program, but why it is preferred, which is 
particularly important. Carefully designed 
and implemented focus group research has the 
potential for providing valuable information 
on important communication elements (such 
as appeal and perceived usefulness) and 
establishes the opportunity for investigators to 
probe for detail that might not be available 
through more quantitative methods. Focus 
groups can be particularly useful in identifying 
unsuccessful approaches before significant 
effort and money are expended on them. As 
with personal interviews, focus groups should 
be taped and transcribed verbatim and/or 
video taped. Video taping, however, is costly 
and the benefits should be weighed against the 
costs. The focus group process is discussed in 
greater detail in the Community How To 
Guide on Needs Assessment and Strategic 
Planning. 

Participant Observation 

Participant observation involves the evaluator 
participating in the event being observed. For 
instance, if the project is evaluating the 
effectiveness of its media outreach efforts, the 
evaluator may attend a news conference to 
determine if the event was successful. What 
the evaluator may look for are any barriers 
that prevent people from participating, the 
smoothness of the operation, the level of 
enthusiasm of participants, the areas of success 
and the areas of weakness. The number of 
events to observe is based on the objective of 
the evaluation. Participant observation can 
be direct, where people know they are being 
observed or unobtrusive where the observer 
does not directly participate in the event or 
activity. 



PLininynEVflLuiin 

1. Write a statement defining the 
purpose(s) of the evaluation. An 

unfocused evaluation cannot accomplish its 
intended goal. 

A statement defining the evaluation purpose 
for an underage drinking prevention project 
may be the following: 

To learn whether the enforcement, prevention 
and education and public policy initiatives 
undertaken by our project are changing the 
attitudes and behavior of both youth and 
adults in our community toward underage 
drinking. 

In some cases, the evaluation may be focused 
on one or more of the activities of the overall 
underage drinking prevention project. For 
instance: 

To learn whether increased compliance checks 
have had a measurable impact on sales to 
minors in the community. 

2. Determine budgetary and other 
resource needs 

The type of evaluation to be conducted will 
be determined by the amount of funding and 
other resources that are available. For 
instance, if the organization does not have 
enough money to hire an independent, 
outside evaluator, then other methods, such as 
finding a college or university that will donate 
these services, would be part of the plan. If 
the plan calls for surveys or focus group 
meetings, is there a sufficient number of 
people in the coalition who can assist with 
these tasks. 
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3. Define the target population. 

The target population can vary depending on 
the objective. In the first example above, the 
target population would include all underage 
youth and adults. The target population for 
the second example would be liquor licensees 
and their employees. 

4, Write down the type of information to 
be collected. 

In the Community How to Guide on Needs 
Assessment and Strategic Plannings underage 
drinking prevention organizations are urged to 
conduct a careful needs assessment and then 
to use the findings as the basis for their 
strategic plan. This information can also serve 
as a baseline and be the staning point for an 
evaluation plan. 

For instance, using the first example of the 
evaluation plans purpose, the information to 
be collected would include the following: 

■ Demographics - the numbers of adults 
and youth under 21 in the community 
(youth ages 15-20, for instance). 

■ Focus/discussion groups with key 
community leaders, public officials, 
parents, youth, etc. 

■ Surveys of youth and household surveys of 
adults. 



baseline for evaluation efforts. It is essential that 
projects have a baseline. The baseline documents 
the situation or problem before a project or 
activity is implemented. Once implementation 
has occurred, the projea can then compare the 
evaluation results to determine whether the effort 
had any affea. Without a baseline, it would be 
impossible to show actual improvement. 

The second example involving the effectiveness of 
compliance checks would also require a baseline, 
which would include a pre- and post-test 
following the enforcement action. The pre-test 
would determine the number of retailers who sell 
to minors and the post-test would determine how 
many sold to minors after the compliance check 
program was initiated. 

5, Choose the type of evaluation to be 

used, 

■ Formative evaluation - Determines 
whether program plans, procedures, 
activities, materials, and modifications will 
work as planned. 

■ Process evaluatior- Determines whether 
programs procedures are working as 
planned. 

■ Outcome evaluation - Determines the 
programs progress toward immediate and 
intermediate goals. 



■ Data showing an increase or decrease in 
enforcement efforts. 

■ Stories in the newspaper, on radio and 
television concerning underage drinking. 

■ Passage of stronger underage drinking laws. 




All of this information should be in the coalition 
or organizations needs assessment, which serves as 
both the basis of the strategic plan and the 
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■ Impact evaluation - Determines whether 
the programs has met is ultimate goal of 
reducing underage drinking. 

6, Determine what methods will be used 
for collecting the information, 

■ Quantitative Methods - Tracking the 
Program (number of people who participated 
or items distributed) surveys, and data. 
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■ Qualitative Methods - Personal interviews, 
focus groups, participant observation 

7. Collect the information and analyze the 
results* 

Usually the individual who analyzes the data is 
a professional evaluator who has the ability to 
interpret what was learned from the 
information and data. A college or university 
professor or graduate student, a state or local 
health department evaluator or an evaluator 
from the state traffic safety or substance abuse 
prevention and treatment agency may be able 
to help. Advertising, marketing and public 
relations agencies may also conduct 
evaluations of their campaigns and may be 
willing to contribute their expertise. 

8* Write an evaluation report describing 
the evaluation results* 

The report can be simple or complex, 
depending upon the needs of the coalition or 
organization. If the evaluation is being used 
to justify further funding, then the 
information that is collected and analyzed 
should reflect the elements in the original 
funding request. In addition, the final report 
should be in form that is consistent to what is 
being required by the funding agency or 
organization. 

The appendix of this guide includes Appendix 
#2 an Evaluation Plan Worksheet to assist 
coalitions and organizations in developing their 
evaluation plan. 



HUG nil EVALUATOR 

To obtain a truly objective view of the underage 
drinking prevention projects effectiveness, it is a 
good idea to hire an outside evaluator. An 
outside evaluator has no history with the 
organization and can offer a new perspective and 
provide fresh insights. The people who are 
actually planning and implementing a program 
often “cannot see the forest for the trees” and 
need an outside perspective. The hiring of an 
outside evaluator, however, costs money and it is 
important that the coalition or organization 
recognize this fact and insure there is sufficient 
funding to complete the task. In some cases, the 
cost of evaluation is part of an overall budget 
request and is usually 20-25% of the total 
amount of the funding request. 




When selecting an outside evaluator, consider the 

following: 

■ Professional training and experience — What 
is their background and what other projects 
have they evaluated? 

■ Style - Does the evaluator consider himself or 
herself to be in partnership with the project 
staff or a neutral observer? 

■ Specialty — Does the evaluator specialize in 
research, data gathering, testing program 
materials, media relations, documenting the 
final results of a program, or has experience in 
a variety of areas? 



When looking for an individual to do qualitative 
evaluation, make sure they have the following 
qualifications: 



■ Has interview and personal observation 
experience. 
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The National Center for Injury Prevention 

and Control offers the following 

information on the characteristics of a 

suitable evaluator: 

■ Is not directly involved in the 
development or running of the program 
being evaluated. 

■ Is impartial about evaluation results (i.e., 
has nothing to gain by skewing the 
results in one direction or another). 

■ Will not give in to any pressure by staff 
or coalition members to produce 
particular findings. 

■ Will give staff the full findings (i.e., will 
not gloss over or fail to report certain 
findings for any reason). 

■ Has experience in the type of evaluation 
needed. 

■ Has experience with similar programs. 

■ Communicates well with key personnel. 

■ Considers programmatic realities (e.g., a 
small budget) when designing the 
evaluation. 

■ Delivers reports and protocols on time. 

■ Relates to the program. 

■ Sees beyond the evaluation to other 
programmatic activities. 

■ Explains both benefits and risks of 
evaluation. 

■ Educates program personnel about 
conducting evaluation, thus allowing 
future evaluations to be done in house. 

■ Explains material clearly and patiently. 

■ Respects all levels of personnel. 




■ Remains neutral and does not show what they 
feel or think about anyone’s response to a 
question. 

■ Good at getting people to participate and 
comment. 

■ Has experience in not allowing one person or 
a few people to dominate the conversation. 

■ Knows how to summarize comments and 
present the result in an easy to understand 
way. 

In some cases, coalitions and organizations may 
not have the resources for professional evaluators 
to perform qualitative evaluation. There may, 
however, be individuals in the coalition who have 
these skills and can perform the tasks just as well. 

In this guide see, Appendix #3 Questions for 
Personal Interviews; Appendix #4 Questions 
for Focus Groups, and Appendix #5 Participant 
Observation Form. 
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conausion 

This guide is designed to explain the evaluation 
process in such a way that it is no longer an 
afterthought in an organizations planning 
process, but a critical first step that can help the 
coalition or organization continue to be 
successful. Following are some conclusions on 
evaluation. 

■ Evaluation does not need to be difficult. 

In order to obtain useable results, keep the 
evaluation as simple and straightforward as 
possible. Resist anyone who tries to expand 
the focus or complicate the design. Keep the 
level of evaluation consistent with the size of 
the project and the objectives you are trying to 
meet. 

■ It does not have to be expensive. A 
project can learn a great deal if they do 
the following: 

■ Conduct a comprehensive needs assessment 
(See Community How To Guide on Needs 
Assessment and Strategc Planning at the 
beginning of the project. 

■ Develop a strategic plan with measurable goals 
and objectives that are based on the 
information from the needs assessment. 



■ Do periodic “mini” needs assessments during 
project implementation to determine whether 
the program is on target and meeting its goals 
and objectives. 

Take advantage of the resources that exist in the 
community. The project might be able to 
convince a university professor or another 
professional from outside the group to assist with 
an evaluation. If an outside evaluator is hired, the 
project can recruit volunteer data collectors from 
members of the coalition or organization. Work 
with the evaluator to identify activities on which 
you can economize, and which areas are worth 
spending a litde extra. 



■ Investing in evaluation can save time 
and dollars over the long hauL 

With the information learned from a 
thorough evaluation, the project can focus 
resources on the most critical problems and 
the most effective countermeasures. The 
project will also be able to adjust programs 
mid-stream to improve effectiveness. And 
most importantly, the project will be much 
more likely to convince funding sources that 
their dollars have been well spent, which 
means that the project is a good investment 
for the future. 
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TRIldGfOIIOl 

What is being counted: 

□ Contact with individual via telephone, fax, mail, e-mail or meeting 

□ Meetings held 

□ People reached by public information and education campaign 

□ Project events, activities or training sessions 

□ People reached through project events, activities or training sessions 

□ Materials distributed 

□ Data (Use data checklist in Community How To Guide on Needs Assessment and Strategic Planning 



CONTACT DESCRIPTION 


Number of 
Contacts 


Date(s) 


Telephone calls 






Fax messages 






E-mail messages 






Meetings 






Number of requests for information 






MEETING DESCRIPTION 


Number of 
Times 


Date(s) 


Coalition or organization meeting 






One-on-one meeting 






Meeting with other community groups 






Meeting with elected official 






Meeting with potential funder 






Meeting with potential new members 






Meeting with key community group, e.g., enforcement, schools, parents, etc. 






Meeting with youth 






Meeting with member of the media 






Other 
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Tracking Form continued 



PUBUC INFORMATION AND EDUCATION CAMPAIGN 


Numbers 

Reached 


Date(s) 


News Story on television or radio 






News story in newspaper 






Public presentation, e.g., speaking before students, parents, civic group, etc. 






Other 






EVENT DESCRIPTION/NUMBERS REACHED 


Number of 
Times 


Date(s) 


News conference 






Attendance at legislative event (rally, hearing, etc.) 






Parent training session 






Youth training session 






Conference 






Other 






MATERIAL DISTRIBUTION 


Number of 
Items Distributed 


Date(s) 


Brochure 






Poster 






Specialty item e.g., key chain, stickers, pens, pencils, etc 






Radio public service announcement 






Television public service announcement 






Campaign kit 






Tee shirts 






Report 






Booklet 






Other 
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EVflLymiooPLiyyopm 



1. What is the purpose or objective of the evaluation? 






2. Define the target population (Example: youth, adults, enforcement, retailers, policy makers, etc.) 




3. Determine the types of information to be collected 

(should match information collected for the baseline report). 

a. Demographics 

b. Data 

(1) Highway Safety (motor vehicle crashes, DWI arrests, etc.) 

(2) Liquor Law Violations 

(3) Sales to Minor Violations 

(4) School data 

(5) Criminal Justice data (convictions) 

(6) Injury/Fatality data (non-motor vehicle) 

(7) Treatment 

(8) Other 

c. Surveys 

(1) Youth 

(2) Adult 

d. Focus/discussion groups 

(1) Youth 

(2) Parents 

(3) Law Enforcement/Alcohol Beverage Control 

(4) Educators 

(5) Elected officials 

(6) Substance abuse prevention/treatment specialists 

(7) Health department 

(8) Faith community 

(9) Alcohol Retailers 

(10) Businesses 

(11) Media 

(12) Other 

e. Media review 

£ Public policy review 
g. Other 


YES 


NO 
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Evaluation Plan Worksheet continued 



4 . Evaluation methods to be used 

a. Formative evaluation 

(1) Description 

(a) Quantitative methods to be used 

(b) Qualitative methods to be used 

(2) When it will be conducted 

b. Process evaluation 

(1) Description 

(a) Quantitative methods to be used 

(b) Qualitative methods to be used 

(2) When it will be conducted 

c. Impact Evaluation 

(1) Description 

(a) Quantitative methods to be used 

(b) Qualitative methods to be used 

(2) When it will be conducted 

d. Outcome Evaluation 

(1) Description 

(a) Quantitative methods to be used 

(2) When it will be conducted 

5. Elements of the Evaluation Report 
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SIlfOPLtPEPiOnflUnTERVIEllKjyOTIOnS 

Questions for Adults/Parents 

1. Do you think underage drinking is a serious problem in this community? Probe: Why? Why not? 

2. Do you know youths under the age of 21 who use alcohol? 

3. Do you know of parents or adults who permit youths under the age of 21 to consume alcohol in their homes? 

4. Where do you think minors under age 21 obtain alcohol? 

5. Under what circumstances is it acceptable for an adult to provide alcohol to minors under age 21? 

6. What forms of advertising do you think influence alcohol use among minors under age 21? 

7. If you were aware of a minor under the age of 21 who was consuming alcohol, what would you do? 

8. What resources are available in your community that address alcohol use among minors under age 21? 

9. Would you favor new and/or stifFer penalties for parents who serve alcohol to minors under the age of 21? 

What about peers over 21 who purchase alcohol for youth under the age of 21? 

What about bars/ restaurants/liquor stores that sell to minors under age 21? 

10. Do you favor drivers license suspension or revocation for minors under age 21 who violate underage drinking laws? 

1 1. Who do you think is most responsible for solving the underage drinking problem? 

Questions for Youth 

1 . How old are you? 

2. Have you ever had alcoholic beverages like beer, wine, wine coolers or liquor? 

3. How often do you drink alcohol? 

4. Do you ever have five or more drinks of alcohol at a time? If “Yes”, have you done this in the last month? 

5. Have you ever driven under the influence of alcohol? Been a passenger in a car with a drunk driver? 

6. Do your parents permit you to drink alcohol in your home? Do your parents know how much you drink? 

7. Have you ever purchased alcohol without an ID? Ever successfully used a fake ID to purchase alcohol? 
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Personal Interview Questions continued 

8. Why do you think underage youth drink alcohol? 

9. Do you think alcohol use by underage youth is a serious problem? Probe: Why? Why not? 

10. Where do underage youth obtain alcohol? Probe: Home, liquor store, bar/ restaurant, friends and relatives. 

11. What would you recommend as a solution to the underage drinking problem? 

Questions for Law Enforcement 

1. Do you think underage drinking is a serious problem in this community? 

2. Do you think law enforcement officers aggressively enforce underage drinking laws? Probe: Why? Why not? 

3. Are there any barriers to effective enforcement of underage drinking laws? If yes, what are they and how can they be overcome? 

4. What additional support do you need to enforce underage drinking laws? 

5. Do you think the legal consequences for underage drinking offenses are sufficient? 

6. Who do think is most responsible for the underage drinking problem? 

7. Does your agency provide alcohol education in the schools? If yes, do you think this is effective? 

8. Do you think judges and the juvenile justice system are doing a good job with respect to underage drinking violations? If yes, 
why? If no, why? 

9. How are underage drinking violations currently handled by the judges, the juvenile justice system and the courts? 

10. What improvements would you recommend in the way underage drinking violations are handled by judges, the juvenile 
justice system and the courts? 

1 1. What is your recommended solution for the underage drinking problem? 

Questions for Educators 

1. Do you think underage drinking is a serious problem? Probe: Why? Why not? 

2. Does your school have an alcohol policy? If yes, is it enforced? 

3. Is alcohol education part of the school’s curriculum? If yes, is it effective? 

4. Does your school sponsor extra-curricular activities to prevent underage drinking? 
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Personal Interview Questions continued 

5 . Do you think parents understand the seriousness of the underage drinking problem? 

6. What role do you think teachers and counselors can play in reducing underage drinking? 

7. What is the most effective way to keep underage youth from drinking alcohol illegally? 

8. What additional support do schools need in educating youth about the consequences of underage drinking? 

9. Does your school have a mechanism in place for identifying and assisting students who may have alcohol or drug-related 
problems? 

10. Does your school reach out to parents about the problem of underage drinking? 

Questions for Public and Elected Officials 

1 . Do you think underage drinking is a serious problem? Probe: Why? Why not? 

2. Do you think there are sufficient laws to reduce underage drinking? If yes, what are the most effective laws? If no, what 
additional laws need to be passed to reduce underage drinking? 

3 . Do you think the public is aware of current underage drinking laws and regulations? 

4. How would you improve underage drinking legislation in this state? 

5 . What is the role of public and elected officials in reducing underage drinking? 

6. How would you improve local laws, policies or procedures related to underage drinking? 

Questions for Health and Prevention/Treatment Specialists 

1 . Do you think underage drinking is a serious problem? Probe: Why? Why not? 

2. What are the barriers, if any, to reducing underage drinking in this community? 

3 . What is the role of health professionals in reducing underage drinking? Prevention specialists? Treatment personnel? 

4. Do you think the general public understands the underage drinking problem? 

5 . What can be done to solve the problem of underage drinking? 

6. Who is most responsible for solving the underage drinking problem? 
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SIlOlPLE focus GROUP pSTIOnS 

1. Is underage drinking a serious problem in (name of the community/town/county)? 

If yes, why? 

If no, why? 

Probes 

Does anyone know or come in contact with underage youth that drink? 

Has there been an alcohol-related incident (crash, death, injury) involving an underage youth? 

2. Do all youth engage in underage drinking or is it just a few? 

Probes 

Is underage drinking more of a problem for some young people than others? 

Is underage drinking just a common right of passage? 

3. What do you think causes underage drinking? 

Probes 

Is it the fault of parents? 

Is it the youth? 

4. Does the community send mixed messages to youth about underage drinking? 

Probes 

Is there a lot of outdoor alcohol advertising? 

Do adults permit underage drinking? 
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Sample Focus Group Questions continued 

5. What are the barriers to solving the problem of underage drinking? 

Probes 

Who or what would stand in the way of effective solutions? 

What prevents the problem from being solved now? 

6. What are your suggestions for solving the problem of underage drinking? 

Probes 

Should there be more education in the schools? 

Should there be stricter enforcement? 

7. What do you think your agency/organization/institutions role is in addressing the problem of underage drinking? 
Probes 

What kinds of programs or activities does your agency! organization! institution do for youth? 

Does your agency! organizationUnstitution pay enough attention to the problem of underage drinking? 
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Meeting Observation 


YES 


NO 


1 . Who attended the meeting? 






Youth 






Parents 






Law Enforcement 






Alcohol Beverage Control 






Judicial System 






Educators 






Health/Prevention Specialists 






Public Officials 






Faith Community Leaders/Officials 






Business Owners 






Alcohol Retailers 






Other 






2. Who participates in the meetings? 


3. How are decisions made? 


4. How is the leadership structure established? 


5. How is the agenda established? 


6, Are there members that are disruptive? If yes, how are they handled? 


7, Are there specific accomplishments achieved by the end of the meeting? 


8. How is the workload divided? 


9. Is the meeting documented and materials sent to participants? 
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Participant Observation Form continued 



Contact Observation 


YES 


NO 


1. Who is contacted? 






Youth 






Parents 






Law Enforcement 






Alcohol Beverage Control 






Judicial System 






Educators 






Health/Prevention Specialists 






Public Officials 






Faith Community Leaders/Officials 






Business Owners 






Alcohol Retailers 






Other 







2. How is the contact made? 



3 . Who makes the contact? 



4. What is discussed? 



5 . Is there follow up? 



Media Event Observation 



1. What was the topic of the news event? 



2. Where was the event held? Was it a good location? 



3. Who were the speakers? 



4. Was the event well organized? 



5 . What materials were distributed? 



6. Were the materials appropriate, well written and newsworthy? 



7. How was the media contacted? 



8. What media attended? 



9. Was it or was it not a successful event? 




xi 
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Participant Observation Form continued 



Legislative Activity 


1. Who supports the legislation? 


2. What barriers were raised against the legislation? 


3. Was the legislation modified? If yes, how? 


4. Was an advocacy effort mounted for or against the legislation? 
If yes, was it effective? Why, why not? 


5. Was the legislation passed? If yes, was there an effort to educate the public 
and/or youth about the legislation? 


Public Information/Education Campaign 


1. What materials were included in the campaign? 


2. Did the target audience(s) positively react to the campaign? If yes, why? If no, why? 


3. Was there any media involvement in the campaign? 


Law Enforcement Activity 


1 . What was the activity? 


2. Who was the target audience? 


3. What occurred? 


4. What was the reaction of the target population? 


5. Were arrests/citations given? If yes, what was the number? 


6. Was the activity/action successful? If yes, why? If no, why? 


7. Did the media cover the enforcement action? 


8. What was the disposition of the citations/tickets/arrests? Fines? Jail? License suspensions? 
Community service? Another sanction? 
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PILOT PROJtdS 

Cities, counties and neighborhoods across America are confronting the problem of underage drinking and its consequences. As 
the professional organization representing the chief highway safety officers from each state, the District of Columbia and the U.S. 
territories, the National Association of Governors’ Highway Safety Representatives (NAGHSR) is committed to helping reduce 
illegal underage alcohol consumption and curb the terrible toll underage drinking takes on our society. 

Underage Drinking Pilot Project 

In March 1995, NAGHSR launched a pilot project on underage drinking funded by the National Highway Traffic Safety 
Administration (NHTSA). The pilot assisted five communities in developing and implementing comprehensive underage drinking 
prevention programs based on a model initiated in the Washington, D.C. area in 1992 at the direction of the U.S. Congress. 

The five NAGHSR pilot sites included Chesterfield County, VA; Travis County (Austin), TX; Omaha, NE; Detroit, MI and Salt 
Lake City, UT. In the first year, the project focused on developing broad-based community coalitions and helping those coalitions 
undertake an extensive needs assessment regarding the nature, extent and consequences of underage drinking in their communities. 

Based on the information obtained during the needs assessment process, each site developed a comprehensive strategic plan that was 
implemented in subsequent years. The strategic plans included goals and objectives that were directly related to specific problems 
identified in the needs assessment. The objectives were specific and measurable so that progress could be tracked over time. 

Each of the communities demonstrated success and four of the five programs continue to operate. The Travis County Underage 
Drinking Prevention Project in Austin, TX was nominated for a national award by the state s highway safety office and the Safe 
and Sober Youth Project in Chesterfield County, VA continues to expand its operations and activities to other counties. Project 
Extra Mile in Omaha, NE and Save Our Youth in Salt Lake City, UT are actively involved in underage drinking efforts funded 
through the Office of Juvenile Justice and Delinquency Prevention (OJJDP). Project Extra Mile has received extensive media 
coverage and was the recipient of a 1999 Award from the National Commission Against Drunk Driving. The programs have 
succeeded in building awareness of the problem in their communities, mobilizing key members of the community to take action 
and changing policies, procedures and laws that directly relate to underage drinking. 

Rapid Response Team 

To build on the success of the demonstration projects, NAGHSR and NHTSA decided in March 1998 to pilot test another 
approach to providing technical assistance to communities working to prevent underage drinking. The revised concept involved 
intense, short-term assistance to communities provided by a team of nationally recognized experts. State highway safety offices 
were invited to nominate communities to receive the technical assistance and NAGHSR and NHTSA chose six sites in various 
states. The “on-site” intensive technical assistance was christened the “Rapid Response Team.” The team provided technical 
assistance for coalitions, which already existed, but were having difficulty designing and implementing underage drinking 
prevention programs. Assistance by the Rapid Response Team gave each site a “jump start” in developing or strengthening 
comprehensive, needs-based underage drinking prevention programs. 
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Pilot Projects continued 



The six communities selected to participate in the pilot, which was conducted from March 1998 to September 30, 1999, included 
Rio Arriba County, NM; Tippecanoe County, IN; Oswego County, NY; Hermantown, MN; Lenoir County, NC, and Prince 
Georges County, MD. Each of the six communities received the following: 

■ Assistance in completing a self-assessment to determine the nature and extent of the underage drinking problem in their 
community. 

■ A three-day site visit by the team of nationally recognized experts, who worked with local advocates and officials to guide them 
in developing a comprehensive program. Members of the team included experts in: 

■ Coalition building, needs assessments and strategic planning 

■ Enforcement and adjudication 

■ Community development 

■ Youth programs and school-based prevention 

■ Public policy 

■ Media Relations 

■ Additional technical assistance was provided for three months from the team by phone, fax and e-mail. 

NAGHSR also led the first phase of technical assistance for the Department of Justice s Office of Juvenile Justice and Delinquency 
Prevention (OJJDP) nationwide program. Enforcing Underage Drinking Laws. In this effort, the NAGHSR team reviewed the 
underage drinking prevention plans for all 50 states and the District of Columbia and provided training for forty-eight states and 
the District of Columbia. The NAGHSR Community How To Guides on Underage Drinking Prevention contain information 
gleaned from the experience of the Washington, D.C., area model (the Washington Regional Alcohol Program and Drawing the 
Line on Underage Alcohol Use), the eleven sites in the two NAGHSR pilot projects, and the OJJDP technical assistance. 



The National Association of Governors' Highway Safety Representatives (NAGHSR) and the National Highway Traffic Safety 
Administration (NHTSA) wishes to thank the leaders in the NAGHSR underage drinking pilot sites and the rapid response pilot 
sites for their leadership and enthusiasm to address underage drinking within their communities. 

The personnel involved in the first pilot effort include the following: Diane Riibe of Project Extra Mile in Omaha, Nebraska; Pat 
Farris, former Executive Director of Children At Risk Today in Chesterfield County, Virginia; Gloria Souhami of the Travis 
County Underage Drinking Prevention Project in Austin, Texas; Cherilynn Uden of Save Our Youth in Salt Lake City, Utah and 
Alma Gale, with the Bureau of Substance Abuse in Detroit, Michigan. 
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Pilot Projects continued 



The pilot of the rapid response initiative involved the following individuals: Juan Roybal of the DWI Prevention Council in Rio 
Arriba County, New Mexico; Sally Mclntire and Debbie Lowe of the Coalition for a Drug-Free Tippecanoe County in Layfayette, 
IN; Jane Murphy, Karen Hoffman and Barbara Canale of Take Charge Coalition in Oswego, New York; Barbara LaRoque of the 
Lenoir County Initiative to Reduce Underage Drinking, Kinston, North Carolina; Chris Olafson and Dave Thompson of the 
Learners At Risk Committee in Hermantown, Minnesota, and Dana Gigliotti of the Prince George’s Highway Safety Task Force 
in Prince George’s County, Maryland. 

The success of the rapid response pilot was due to the involvement of the team of experts who provided valuable assistance and 
expertise. Individuals who participated as part of the Rapid Response Team included Marie Bishop, former Governor’s Highway 
Safety Representative in Idaho; Jim Copple, National Crime Prevention Council; Johnnetta Davis, Pacific Institute for Research 
and Evaluation; Captain Tom Didone, Montgomery County, Maryland Police Department; Nancy Chase Garcia, Garcia 
Consulting (formerly with Center for Substance Abuse Prevention); Andrew Hill, Higher Education Center for Alcohol and 
Other Drug Prevention; Officer William Morrison, Montgomery County, Maryland Police Department; Pat Nechodom, 
University of Utah; Nancy Rea, Drawing the Line on Underage Alcohol Use, Montgomery County, Maryland; Judy Robinson, 
Higher Education Center for Alcohol and Other Drug Prevention; Mary Ann Solberg, Troy Community Coalition; Ed Virant, 
Omaha Public Schools, and Lt. Dick Yost, Phoenix, Arizona Police Department. 

Thanks also goes to the many individuals, coalitions and organizations whose ongoing dedication to reducing underage drinking 
provided an opportunity to test the validity of NAGHSR’s comprehensive approach. Their ideas, enthusiasm and commitment 
demonstrate there is a willingness across this nation to tackle the problem of underage drinking. 
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commyniTynouj TO GUIDE 

The Art of Appropriate Evaluation. 

A Guide for Highway Safety Managers 

National Highway Traffic Safety Administration 

Office of Research and Evaluation 

400 Seventh St., SW 

Washington, D.C. 20590 

202-366-9588 

Fax: 202-366-2766 

Web site: http://www.nhtsa.dot.gov 

Demonstratii^ Your Program’s Worth. 
A Primer on I^aluation Programs To 
Prevent Unintentional Injury 

Centers for Disease Control 

National Center for Injury Prevention and Control 

Mailstop K65 

4770 Buford Highway, NE 

Atlanta, GA 30341-3724 

770-488-1506 

Fax: 770-488-1667 

Web site: http://www.cdc.gov/ncipc 

Guide to Conducting Youth Surveys 

Underage Drinking Enforcement Training Center 

Pacific Institute for Research and Evaluation 

1 1 140 Rockville Pike, 6th Floor 

Rockville, MD 20852 

301-984-6500 

Fax: 301-984-6559 

Web site: http://www.pire.org/udetc 

National Survey of Drunk Driving 
Attitudes, 1997 (DOT HS 808-8M) 

National Highway Traffic Safety Administration 

Office of Research and Traffic Records 

400 Seventh St., SW 

Washington, D.C. 20590 

Fax: 202-366-7096 

Web site: http://www.nhtsa.dot.gov 



Understanding Evaluation: The Way 
to Better Prevention Programs, 1993 

By Lana Muraskin 

U.S. Department of Education 

Office of Elementary and Secondary Education 

Safe and Drug Free Schools 

400 Maryland Avenue, SW 

Washington, DC 20202 

800-USA-LEARN 

Fax: 202-401-0689 

Web site: http://www.ed.gov/offices/OESE/SDFS 

Youth Behavioral Risk Survey 

Centers for Disease Control 
4770 Buford Highway, NE 
Atlanta, GA 30341-3724 
770-488-1506 
Fax: 770-488-1667 

Web site: http://www.cdc.gov/nccdphp/dash/yrbs 

OTHER EVULUiOn RESOURCES 

American Evaluation Association 

PO Box 704 

Point Reyes CA 94956 

888-311-6321 

Web site: http://www.eval.org 

The American Evaluation Association is an 
international professional association of evaluators 
devoted to the application and exploration of 
program evaluation, personnel evaluation, 
technology, and many other forms of evaluation. 
The associations mission is to improve evaluation 
practices and methods, increase evaluation use, 
promote evaluation as a profession and support 
the contribution of evaluation to the generation 
of theory and knowledge about effective human 
action. 
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The Evaluation Exchange 

Harvard Research Project 

38 Concord Avenue, Cambridge, MA 02138 

617-495-9108 

Fax:617-495-8594 

Web site: httpiHgseweb. harvard, edu/ ^ hfrp 

The Evaluation Exchange is an interactive forum 
for the exchange of ideas, lessons, and practices in 
the evaluation of family support and community 
development programs, promoting discussion 
among persons from a variety of organizational 
affiliations and viewpoints. Vol. V, No. 1 1999 of 
the exchange focuses on evaluating programs 
serving children and youth. 

Innovation Network, Inc. (InnoNet) 

1001 Connecticut Avenue, NW, #900 

Washington, D.C. 20036 

202-728-0727 

Fax 202-728-0136 

Web site: http://www.inetwork.org 

The Innovation Network, Inc. (InnoNet) is an 

organization dedicated to helping small- to 

medium-sizednonprofit organizations successfully 

meet their missions. The purpose of their web site 

is to provide the tools, instruction, guidance 

framework to create detailed program plans, 

evaluation plans andfund-raising plans. 

Outcome Measurement 
Resource Network 

United Way of America 
701 N. Fairfax Street 
Alexandria, Virginia 22314-2045 
703-836-7100 

Web site: http://www.unitedway.org 



publications, pricing, and ordering information 
and selected outcome and performance 
measurement initiatives of United Ways, health 
and human service agencies, governmental and 
other nonprofit organizations, and links to other 
internet resources. The network is available for 
use by the general public. 




Board Assessment of the Organization: 
How Are We Doing? 

by Peter Szanton 

National Center for Non-Profit Boards 
1828 L Street, NW, Suite 900 
Washington, DC 20036-5104 
202-452-6262 or 800-883-6262 
Fax: 202-452-6299 
Web: http://www.ncnb.org 

This booklet provides key questions that board 
members and executive directors should ask when 
assessing their organizations performance. It also 
explains how to determine who should perform 
the evaluation, what it should examine, when it 
should be performed, and how it should be 
conducted. 

Empowerment Evaluation: 

Knowledge and Tools for Self- 
assessment and Accountability 

by D.M. Fetterman, S. Kaftarian, and A. 
Wandersman (1996) 

Sage Publications Ltd 
6 Bonhill Street 
London 
EC2A 4PU 






The Resource Networks purpose is to provide 
United Way of Americas (UWA) and other 
organizations’ outcome measurement resources 
and learning. The network includes the following: 
a section on FAQ (Frequently Asked Questions), 
descriptions of UWA outcome measurement 
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United Kingdom 
Telephone: +44 (0)171 374 0645 
Fax: +44 (0)171 374 8741 
E-mail: online-info@sagepub.co.uk 
Book orders hotline: +44 (0)171 330 1234 
Web site: http://www.sagepub.co.uk 
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Empowerment evaluation — a method for using 
evaluation concepts, techniques and findings to 
foster improvement and self-determination - is 
the focus of this book. After an examination of 
the method as it has been adopted in academic 
and foundation settings, the book looks at the 
various contexts in which empowerment 
evaluation is conducted, ranging from resistant 
environments to responsive environments. 

Critical concerns in empowerment evaluation, 
such as the role of empowerment theory and 
multiple levels of empowerment from individual 
to societal, are then discussed. The book also 
provides tools and technical assistance needed to 
conduct empowerment evaluation. The 
concluding section of the book serves to 
strengthen the links between empowerment 
evaluation and community-capacity building. 

Evaluator’s Handbook 

by Joan L. Herman, Lynn Lyons Morris, and 
Carol Taylor Fitz-Gibbon (1997) 

Sage Publications Ltd 

6 Bonhill Street 

London 

EC2A 4PU 

United Kingdom 

Telephone: +44 (0)171 374 0645 

Fax: +44 (0)171 374 8741 

E-mail: online-info@sagepub.co.uk 
Book orders hotline: +44 (0)171 330 1234 
Web site: http://www.sagepub.co.uk 

This volume is at the core of the Program 
Evaluation Kit. It takes a step-by-step approach to 
evaluation, using non-technical language to 
explain procedures to novice evaluators. This 
edition reflects the current emphasis on 
continuous evaluation throughout the process of 
program development. New references and the 
inclusion of evaluation standards are also a 
feature. The Evaluators Handbook is illustrated 
with examples, suggestions, worksheets and 







sample forms for the readers own use. At 
appropriate points, it refers readers to other 
volumes in the Kit for further information. 

Handbook of Practical Program 
Evaluation (1994) 

by Joseph S. Wholey, Harry P. Hatry, and 

Kathryn E. Newcomer 

Jossey-Bass Publishers 

350 Sansome Street 

San Francisco, CA 94104 

888-378-2537 800-956-7739 

Web site: http ://www.jossey bass, com 

Experts in the field of program evaluation outline 
efficient and economical methods of assessing 
program results and identifying ways to improve 
program performance. From simple evaluation to 
more thorough examinations, the authors describe 
the nuts and bolts of how to create evaluation 
design and how to collect and analyze data in a 
way that will result in low cost and successful 
evaluations. 




“Promising Aj^roaches In The 
Prevention ofUnderage Drinking” — 

A Final Report” 

National Association of Governors’ Highway 
Safety Representatives (NAGHSR) 

750 1st Street, NE, Suite 720 
Washington, DC 20002 
202-789-0942 
Fax: 202-789-0946 
Web site: http://www. naghsr. org 

The document, jointly developed by NAGHSR 
and The National Association of State Alcohol 
and Drug Abuse Directors (NASADAD) for The 
National Highway Traffic Safety Administration 
and The Center for Substance Abuse Prevention, 
contains case studies of state-wide activities to 
prevent underage drinking in nine states. Case 
studies include: North Dakota — Alternative 
Activities Through the Community Traffic Safety 
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Program Network; New York — Athletes 
Helping Athletes, Inc. of Long Island; New 
Jersey — Smoke and Alcohol-Free Residence 
Halls and Campus Entertainment Centers; 
Massachusetts — Working with Servers and 
Sellers to Restrict Access; Virginia — Combating 
Fraudulent Identification Use; Washington — 
Talking to Your Kids About Alcohol; Maryland 
— Maryland Underage Drinking Prevention 
Coalition; California — Teenwork and Ohio — 
None for under 2 1 . The report summarizes 
common themes and includes contact names and 
NAGS HR and NASADAD membership lists. 

Self-Assessment for Nonprofit Boards 

by Larry Slesinger 

National Center for Non-Profit Boards 
1828 L Street, NW, Suite 900 
Washington, DC 20036-5104 
202-452-6262 or 800-883-6262 
Fax: 202-452-6299 
Web: http://www.ncnb.org 

This book shows boards how to evaluate their 
overall performance in a number of areas as well 
as each members contribution to the board s 
work. It includes a 50-page users guide and 15 
copies of a 20-page questionnaire for each 
member to fill out. Additional questionnaires are 
available separately. 



V(^K. Kellogg Foundation Evaluation 
Handbook 

WK. Kellogg Foundation 

One Michigan Avenue East 

Battle Creek, Michigan 49017-4058 

616-968-1611 

Web site: 

http://www.wkkf.org/Publications/evalhdbk/def 

aulthtm 

This handbook is guided by the belief that 
evaluation should be supportive and responsive to 
projects, rather than become an end in itself It 
provides a framework for thinking about 
evaluation as a relevant and useful program tool. 

It is written primarily for project directors who 
have direct responsibility for the ongoing 
evaluation of WK. Kellogg Foundation-funded 
projects. However, it is the hope of the 
foundation that project directors will use this 
handbook as a resource for other project staff who 
have evaluation responsibilities, for external 
evaluators, and for board members. 
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